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Student Understanding of Basic Mathematical Principles”^ 

The findings from this review do not refiect the fuii body of research evidence on POWERSOURCE®. 


What is this study about? 

The study examined the impact of POWERSOURCE®, 
an intervention consisting of formative assessments, 
instructional resources, and professional develop- 
ment designed to help teachers provide individual 
instruction to their students in Algebra I. This study 
took place in seven districts in Arizona and California 
during the 2007-08 school year. 

The design for this study used two different types of 
randomization. In three districts, 26 sixth-grade teach- 
ers were randomly assigned to use POWERSOURCE® 
and 23 other sixth-grade teachers to a comparison 
group. In the four remaining districts, researchers 
randomly assigned 1 7 schools either to use POWER- 
SOURCE® or to be in the comparison condition. 

The POWERSOURCE® intervention targets four 
domains: (a) rational number equivalence, (b) proper- 
ties of arithmetic, (c) principles for solving linear equa- 
tions, and (d) applications of core principles. Teachers 
in the intervention condition used approximately eight 
class periods (two per domain) for student assess- 
ment and instruction with POWERSOURCE®. Three 
assessments were used for each domain. In addition 
to the formative assessments, the POWERSOURCE® 
intervention included resources for teachers that 
explained the results of the assessments and how 
they could be used to increase student learning. 
Teachers in the comparison condition were offered 
a different (non-POWERSOURCE®) professional 


WWC Rating 


The research described in 
this report for the sampie that 
randomiy assigned teachers to 
condition meets WWC evidence 
standards without reservations 

Strengths: The study includes a well-implemented 
randomized controlled trial of teachers with low levels 
of sample attrition for the main outcome and the 
three subscale outcomes (properties of arithmetic, 
rational number equivalence, and solving equations). 

Cautions: The research on the sample that randomly 
assigned schools to condition does not meet WWC 
evidence standards because there were high levels 
of attrition, and the analytic samples were not 
demonstrated to be equivalent at baseline. 


development program from the National Center 
for Research on Evaluation, Standards, and 
Student Testing. 

Researchers assessed the effectiveness of POWER- 
SOURCE® by comparing student performance on a 
mathematics achievement test. Results were pre- 
sented for a composite score and for the three sub- 
scales that comprise the main achievement outcome. 
The outcomes assessed dimensions of student 
performance in algebra and were constructed from 
a range of national, state, and local assessments. 
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What did the study find? 

The study reported that students of teachers 
who were assigned to use the POWERSOURCE® 
intervention performed better on questions about 
properties of arithmetic. The WWC did not confirm 
this finding to be statistically significant after adjust- 
ing for the clustering of students within teachers as 
the unit of assignment. 


Features of POWERSOURCE® 


POWERSOURCE® is an intervention consisting of 
formative assessments, instructional resources, and 
professional development designed to help teachers 
provide individual instruction to their students. The 
POWERSOURCE® intervention targets four domains: 
(a) rational number equivalence, (b) properties of 
arithmetic, (c) principles for solving linear equations, 
and (d) applications of core principles. 

Three assessments are administered in each domain 
to help guide instruction. The first assessment has 
eight to 1 0 items on a math concept and takes 
about 15-20 minutes, followed by an analysis of 
results and focused instruction to address problem 
areas identified by the assessment. The second 
assessment, which has four to five items, takes about 
15 minutes and is followed by further instruction on 
applications of the concept. The third assessment, 
also four to five items, takes about 15 minutes and 
is followed by Instruction, if necessary. 

In addition to the formative assessments, teachers 
receive resources that provide explanation of the 
assessment results and professional development 
on how to use the results to inform instruction. 
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Appendix A: Study details 

Phelan, J., Choi, K., Vendlinski T., Baker, E. L., & Herman, J. L. (2009). The effects of POWERSOURCE 
intervention on student understanding of basic mathematicai principles. CRESST Report 763. 
Los Angeles, CA: National Center for Research on Evaluation, Standards, and Student Testing, 
University of California, Los Angeles. 


Setting 

The study was conducted in one school district in Arizona and six schooi districts in Caiifornia. 

Study sample 

Study authors recruited seven schooi districts to participate during the 2007-08 schooi year. 
Across ail districts, 55% of the students were not proficient in math, and 10% to 26% of the 
students were English language learners. In three districts, 49 sixth-grade teachers were 
randomized to the intervention and comparison groups within ten schoois in Caiifornia. At the 
time of random assignment, there were 2,052 students in the intervention condition and 1 ,473 
students in the comparison condition. The sample of students with pretest and posttest data 
inciuded 1 ,496 students in the intervention condition and 1 ,120 students in the comparison 
condition. 

in the four other districts, 17 schools were randomized within districts to the intervention and 
comparison groups, containing 2,078 students at the time of random assignment. The sampie 
of students with pretest and posttest data inciuded 842 students in the intervention schools 
and 633 students in the comparison schoois. This WWC report considers each randomly 
assigned sample separately, and only the sample of schools using within-schooi random 
assignment meets WWC standards without reservations. 

Intervention 

group 

Teachers in the intervention group used approximateiy eight class periods for student assess- 
ment and instruction with POWERSOURCE®. The POWERSOURCE® intervention targets four 
domains related to success in Algebra 1: (a) rational number equivalence, (b) properties of arith- 
metic, (c) principies for solving iinear equations, and (d) applications of core principles. For each 
domain, three assessments were administered to heip determine the content to be taught. 

Comparison 

group 

Teachers in the comparison condition were offered a different {non-POWERSOURCE®) profes- 
sional development program. 

Outcomes and 
measurement 

The main outcome measure was administered at the beginning and end of the study year and 
was referred to as the transfer measure. The transfer measure was constructed with items 
from other assessments, including the Third internationai Mathematics and Science Study 
(TiMSS), the National Assessment of Educational Progress (NAEP), the Qualifications and 
Curricuium Authority (QCA) Key Stage 3 exam, the Programme for international Student 
Assessment (PISA), and a test from a pilot district. The transfer measure has three subscaies: 
(a) properties of arithmetic, (b) rationai number equivaience, and (c) solving equations. For a 
more detailed description of these outcome measures, see Appendix B. 
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Support for 
implementation 


Reason for 
review 


Teachers in the intervention group were told about the study objectives and formative 
assessment prior to implementing POWERSOURCE®. They were also given suggestions 
for analysis of student data and how it could be used to inform instruction. Follow-up 
meetings were held after each of the first three POWERSOURCE® sessions with students. 
In these meetings, teachers were shown an analysis of the incorrect response patterns 
in an assessment of students in their district. In total, teachers received approximately 
nine hours of professional development to support implementation. 

This study was identified for review by the WWC because it was supported by a grant 
to the University of California, Los Angeles (Principal Investigator: Eva Baker) from the 
National Research and Development Centers at the Institute of Education Sciences (lES). 
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Appendix B: Outcome measures for the mathematics achievement domain 


Mathematics achievement 


Mathematics achievement 


Transfer measure 

The transfer measure was constructed using items from severai assessments, including the TiMSS, the NAEP, 
the QCA Key Stage 3 exam, the PiSA, and a test from a pilot district. The items on the transfer measure were 
chosen because they were related to the topics in POWERSOURCE® but were not the same as the questions 
in POWERSOURCE® assessments. Cronbach’s alpha for this measure is 0.86. 

Properties of arithmetic 

This assessment is a subscale of the transfer measure and contains five items about the distributive property 
in arithmetic. 

Rationai number equivaience 

This assessment is a subscale of the transfer measure and contains 11 items about the relative magnitudes 
of fractions. 

Solving equations 

This assessment is a subscale of the transfer measure and contains 14 items about solving equations. 
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Appendix C: Study findings for the mathematics achievement domain 


Mean 

(standard deviation) WWC calculations 


Domain and 
outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Mathematics achievement 

Transfer measure 

Sixth-grade 

10 schools/ 

17.54 

17.15 

0.39 

0.07 

-h3 

nr 


students in 

2,616 

(6.12) 

(5.38) 






schools using students 
within-school 
random 
assignment 

Domain average for mathematics achievement 0.07 +3 Not 

statistically 

significant 

Table Notes: For mean difference, effect size, and improvement index vaiues reported in the tabie, a positive number favors tbe intervention group and a negative number favors tbe 
comparison group. Tbe effect size is a standardized measure of tbe effect of an intervention on student outcomes, representing tbe change (measured in standard deviations) in an 
average student’s outcome that can be expected if tbe student is given tbe intervention. Tbe improvement index is an aiternate presentation of tbe effect size, reflecting tbe change 
in an average student’s percentile rank that can be expected if the student is given the intervention. The statistical significance of the study’s domain average was determined by the 
WWC; the study is characterized as having an indeterminate effect because the effect is neither statistically significant nor substantively important, nr = not reported. 

Study Notes: The study used a hierarchical linear model (HIM) to combine data from the analysis of schools using both within-school random assignment as well as between- 
school random assignment. The model as described does not appropriately estimate the difference in outcomes of the intervention and comparison students in the pooled sample. 
As a result, the WWC considers the analysis of the sample of the schools using within-school random assignment and the sample of schools using between-school random 
assignment separately. The analysis of the sample of schools using within-school random assignment meets WWC standards without reservations, and the analysis of the sample 
of schools using between-school random assignment does not meet WWC standards. Consequently, the WWC has used the unadjusted means reported in the study for all WWC 
calculations reported above. A correction for clustering was needed and resulted in a WWC-computed p-value of 0.62 for the transfer measure; therefore, the WWC does not find 
the result to be statistically significant. Subtest results for the transfer measure are presented in Appendix D. The WWC calculated the intervention group mean by adding the 
difference-in-differences adjusted estimate of the average impact of the program (i.e., difference in mean gains between the intervention and comparison groups) to the unad- 
justed comparison group posttests means. Please see the WWC Procedures and Standards Handbook (version 2.1) for more information. 
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Appendix D: Supplemental findings by domain 





Mean 

(standard deviation) 

WWC calculations 


Domain and 
outcome measure 

Study 

sample 

Sample 

size 

Intervention Comparison 
group group 

Mean Effect Improvement 

difference size index 

p-value 


Mathematics achievement 


Properties of arithmetic 

Sixth-grade 
students in 
schoois using 
within-schooi 
random 
assignment 

10 schools/ 
2,616 
students 

1,39 

(1.41) 

1.12 

(1.00) 

0.27 

0.22 

-f9 

nr 

Rationai number 
equivaience 

Sixth-grade 
students in 
schools using 
within-schooi 
random 
assignment 

10 schools/ 
2,616 
students 

6,45 

(2.18) 

6.29 

(2.23) 

0,16 

0,07 

-f3 

nr 

Soiving equations 

Sixth-grade 
students in 
schools using 
within-schooi 
random 
assignment 

10 schools/ 
2,616 
students 

7.75 

(2.97) 

7.54 

(2,81) 

0.21 

0.07 

-f3 

nr 


Table Notes: For mean difference, effecf size, and improvement index vaiues reported in the tabie, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on student outcomes, representing the change (measured in standard deviations) 
in an average student’s outcome that can be expected if that student is given the intervention. The improvement index is an aiternate presentation of the effect size, refiecting the 
change in an average student's percentiie rank that can be expected if the student is given the intervention, nr = not reported. 

Study Notes: The study used an HLM to combine data from the anaiysis of schoois using both within-schooi random assignment as weii as between-schooi random assignment. 
The modei as described does not appropriateiy estimate the difference in outcomes of the intervention and comparison students in the pooied sampie. As a resuit, the WWC 
considers the anaiysis of the sampie of the schoois using within-schooi random assignment and the sampie of schoois using between-schooi random assignment separateiy. The 
anaiysis of the sampie of schoois using within-schooi random assignment meets WWC standards without reservations, and the anaiysis of the sampie of schoois using between- 
schooi random assignment does not meet WWC standards. Consequentiy, the WWC has used the unadjusted means reported in the study for aii WWC caicuiations of subtest 
resuits reported above. A correction for ciustering was needed and resuited in WWC-computed p-vaiues of 0.11, 0.59, and 0.59 for the properties of arithmetic, rationai number 
equivaience, and soiving equations subtests, respectiveiy; therefore, the WWC does not find any of the subtest resuits to be statisticaiiy significant. The resuit of the main outcome 
from which these subtests are derived is presented in Appendix C. The WWC caicuiated the intervention group mean by adding the difference-in-differences adjusted estimate of 
the average impact of the program (i.e., difference in mean gains between the intervention and comparison groups) to the unadjusted comparison group posttests means. Piease 
see the WWC Procedures and Standards Handbook (version 2.1) for more information. 
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Endnotes 

■' Single study reviews examine evidence published in a study (supplemented, if necessary, by information obtained directly from the 
author[s]) to assess whether the study design meets WWC evidence standards. The review reports the WWC’s assessment of whether 
the study meets WWC evidence standards and summarizes the study findings following WWC conventions for reporting evidence on 
effectiveness. This study was reviewed using the single study review protocol, version 2.0. 

Recommended Citation 

U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. (2013, April). 

WWC review of the report: The effects of POWERSOURCE® intervention on student understanding of 
basic mathematical principies. Retrieved from http://whatworks.ed.gov. 
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Glossary of Terms 

Attrition 


Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Improvement index 


Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Single-case design 
(SCD) 

Standard deviation 


Statistical significance 
Substantively important 


Attrition occurs when an outcome variable is not avaiiabie for aii participants initiaiiy assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If intervention assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review if it falls within the scope of the review protocol and uses either 
an experimental or matched comparison group design. 

A demonstration that the analysis sample groups are similar on observed characteristics 
defined in the review area protocol. 

Along a percentile distribution of students, the improvement index represents the gain 
or loss of the average student due to the intervention. As the average student starts at 
the 50th percentile, the measure ranges from -50 to +50. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which subjects are assigned 
to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which investigators randomly assign 
eligible participants into intervention and comparison groups. 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 

The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample are spread out over a large range of values. 

Statistical significance is the probability that the difference between groups is a result of 
chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% (p < 0.05). 

A substantively important finding is one that has an effect size of 0.25 or greater, regardless 
of statistical significance. 


Please see the WWC Procedures and Standards Handbook (version 2.1) for additional details. 
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